A Framework For Supporting Load Shedding in Data Stream Management Systems

The arrival rate of tuples in a data stream can be unpredictable and bursty. Many stream-based applications have Quality of Service (QoS) requirements that need to be satisfied by the underlying stream processing system. In order to avoid violating predefined QoS requirements during temporary overload periods, a load shedding strategy is necessary and critical for a data stream management system. In this paper, we propose a framework and techniques for a general load shedding strategy by dynamically inserting load shedders into query plans or deactivating existing shedders based on the estimation of current system load. These shedders can drop tuples in either a randomized manner or using user-specified application semantics. Specifically, we address the problem of predicting the system load which implicitly determines when load shedding is needed and how much. We also address the problem of determining the optimal placement of a shedder in a query plan, and the problem of how to distribute the total number of tuples to be dropped among all shedders with the goal of minimizing the total relative errors in the final query results due to load shedding. Finally, we have conducted extensive experiments to validate the effectiveness and efficiency of the proposed load shedding techniques.